Artificial Neural Networks

Artificial Neural Networks (ANNs) are machine learning algorithms that imitate neural function. There is a vast theory of ANNs available these days. To limit the scope and length of this project summary, we will only consider three ANN algorithms, which are typically the first ones introduced when starting to work with ANNs. They are the Perceptron, Adaline, and multi-layer feedforward networks. They are all supervised learning algorithms, meaning that they train their parameters based on a set of pre-classified (or labeled) training data. That is, each element of the training data has a known label associated to it. The goal is to learn how to classify novel input according to a rule learned from the training data. We should acknowledge that calling the Perceptron and Adaline models neural networks may be overstating their complexity. Indeed these models are only formulated based on only a single neuron. Hence they could be viewed as a trivial network, but it may be more conducive to think of them as a primer for building mathematical models of neurons. Their construction introduces some foundational ideas of the theory, and they make it much more manageable to work with more complicated networks, like multi-layer networks. There are ways to accomplish more complicated tasks by preprocessing training data and/or using several Perceptrons/Adaline neurons, but even in these models the neurons still function largely independently (not in a coordinated way inherent to more complicated ANN models). We mention a couple ways to extend these models in the Possible Extensions section. There is an important development in the transition from the Perceptron to the Adaline model. Roughly speaking, the Perceptron learns by using a somewhat ad hoc training rule that can be motivated by a geometric argument. The training is a little cumbersome and is reliant on a linear structure in some ways. This makes it difficult to extend directly to more complicated and nonlinear models. Adaline introduces a shift in point of which, which is to train the neuron by minimizing an error function. This notion of minimization in place of a more geometric argument is much more easily extended to more complicated settings, which can be observed in the construction of the multi-layer networks. Much of the information presented here is taken from Mitchell’s book on machine learning, but several aspects are presented differently and at times more concretely. In particular, the details of the learning rules here are laid out in more detailed but less general terms. This description is also much less comprehensive, which allows a much shorter presentation of the material. This may be of use for those just starting to work with ANNs, but would probably be best used in along with other resources.